Micro-Signatures: The Signatures Hidden in Anomaly Detection Systems

The field of intrusion detection is divided into signature detection and anomaly detection. The former involves identifying patterns associated with known attacks and the latter involves attempting to learn a ‘normal’ pattern of activity and then producing security alerts when behaviors outside of those norms is detected. The ngrams methodology has arguably been the most successful technique for anomaly detection (including for network packet inspection). In this work, we identify a new type of intrusion detection that neither uses typical signatures nor is anomaly based (though it is closely related to both). We generate n-grams from both malicious content and Snort signatures and use sets of these ‘microsignatures’ to identify attacks. This micro-signature capability arises implicitly when the training sets for n-gram anomaly detection systems are scrubbed of malicious content and thus is not new. It was added explicitly by the seminal Anagram network anomaly approach, but was portrayed as a minor enhancement and its effect was not evaluated. In reproducing the Anagram results we find that for our data, the micro-signatures provide the vast majority of the detection capability. What appears on the surface to be an anomaly detection approach achieves most of its effectiveness from a (sometimes merely implicit) signature subsystem. We furthermore find that these micro-signatures enable highly effective standalone detection systems as well as hybrid microsignature/anomaly systems that generalize to multiple attack classes. Our results thus shed new light into the functioning of n-gram anomaly detection systems, reveal the need to evaluate the microsignature contribution within n-gram anomaly research, and open a new avenue of research into how to best use micro-signatures in future detection systems.